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1 MARKING MEDIUM AREA WITH ENCODED IDENTIFIER FOR 

2 PRODUCING ACTION THROUGH NETWORK 

3 This application claims priority under 35 U.S.C. §120 from copending International 

4 Application PCT/US98/20597, filed 30 September 1998, with respect to all shared subject matter. 

5 Field of the Invention 

6 The present invention relates to a marking medium area that has markings encoding an 

7 identifier for producing action through a network. 

8 Background of the Invention 

9 It is well known to link electronic documents by setting up hyperlinks between documents in 

10 HTML format stored at, and transferable between, computers forming nodes of a conventional 

1 1 computer network, e.g. the Internet or an intranet. See Berners-Lee, T J., Cailliau, R., and Groff, J.- 

12 F., The Worldwide Web, Computer Networks and ISDN Systems 25, North-Holland, 1992, pp. 454- 

13 459. 

14 Sodergard, C, Juhola, H., Backstrom, C, and Vainikainen, I., "A Pen Scanner Based 

15 System For An Easy Access to Relevant WWW Hyperlinks", in Proceedings of TAGA '97 

16 Conference, Quebec, CA, 4-7 May 1997, Technical Association for Graphic Arts, 1997, describe a 

17 system in which a pen scanner enters printed anchor words linked to uniform resource locator 

18 (URL) addresses. The scanned text is processed with optical character recognition (OCR), and the 

19 OCR result is processed in an interpretation module to select a correct link anchor from a locally 

20 stored link list that has been downloaded from the publisher's World Wide Web (WWW) server. 

21 After selecting the link anchor, the corresponding URL addresses are determined from the link list 

22 and the WWW pages are retrieved with an Internet browser and displayed on a monitor. 

23 Robinson, P., Sheppard, D., Watts, R., Harding, R., and Lay, S., "A framework for 

24 interacting with paper", EUROGRAPHICS '97, Vol. 16, No. 3, 1997, pp. C-329 to C-334, describe a 

25 framework for preparation and presentation of mixed-media documents using a registry that 

26 associates physical locations on pieces of paper with actions. A DigitalDesk, which has a video 

27 camera mounted above a desk to detect where a user is pointing and to read documents on the 

28 desk and a projector mounted above the desk to project objects onto the work surface and paper 

29 documents, is used to identify pieces of paper and animate them by placing them on the 

30 DigitalDesk. The camera identifies the document and follows the pointer, and associated actions 

31 are identified in the registry and invoked as appropriate with the results being projected back onto 

32 the paper. To identify a document, a page is marked with a unique OCR font identifier that encodes 

33 the location of the directory as a network IP address and an index for the document within the 

34 directory. A document can be printed from the registry, with its unique identifier; a printed 

35 document's page representation is retained in the registry as an immutable copy of its structure. 

36 Paper access to the World Wide Web is possible. Given a URL, information on the associated web 

37 page can be captured in the registry, the page can be printed, and links can be activated by placing 

38 the paper on a DigitalDesk and pointing. The page's identifier and the coordinates of the link are 
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1 looked up in the registry to yield the appropriate activity, and the results are projected back onto the 

2 desk. 

3 Johnson, W., Jellinek, H., Klotz, L., Rao, R., and Card, S., "Bridging the Paper and 

4 Electronic Worlds: The Paper User Interface", INTERCHI '93 Conference Proceedings, ACM, 1993, 

5 pp. 507-512, describe techniques in which documents can contain, in addition to the human- 

6 readable information (e.g. text), printed machine-readable encoded data. The document can, for 

7 example, be a form and the encoded data can be a form identifier. A prototype system can receive 

8 a faxed form and decode the form identifier, locate an appropriate action description file, process 

9 the page image, and provide the information from the page image to the action procedure for the 

10 form. The system automatically creates a new form that points at a newly stored document, to give 

11 the user a way of accessing and interacting with it later, with checkboxes that allow the user to print 

12 or delete it. 

13 US-A-5,661,506 describes a pen and paper information recording system using an imaging 

14 pen. A writing paper has a writing surface and a prerecorded invisible pattern of pixels. Each pixel 

15 contains encoded, optically readable position information that identifies a coordinate position on the 

16 writing surface, and may also include a page number and a pad number. The system includes an 

17 imaging system for providing image signals representing images of pixels near the pen tip. A 

18 processor responds to the image signals and determines and electronically records positions of the 

19 pen tip on the writing surface as markings are made. 

20 US-A-5,486,686 describes machine readable electronic domain definitions of part or all of 

21 electronic domain descriptions of hardcopy documents and/or of part or all of the transforms that 

22 are performed to produce and reproduce such hardcopy documents, encoded in codes that are 

23 printed on the documents. The codes permit the electronic domain descriptions of the documents 

24 and/or the transforms to be recovered more robustly and reliably when the information carried by 

25 the documents is transformed from the hardcopy domain to the electronic domain. Encoded data 

26 embedded in the hardcopy document may include descriptions of the data points for structured 

27 graphics, descriptions of algorithms utilized for performing computations for spreadsheets, 

28 descriptions of hypertext pointer values, descriptions of structural characteristics of an electronic 

29 source document, descriptions of a document editor, descriptions of file name and storage location 

30 of an electronic source document, and descriptions of audit-trail data for an electronic source 

31 document. 

32 Summary of the Invention 

33 The invention addresses problems in obtaining automatic actions through a network. It is 

34 often difficult to obtain an appropriate automatic action such as access to multimedia information or 

35 other information available through a network. This is especially true where the context includes a 

36 physical object such as a hardcopy document, and the action should be appropriate to the object. 

37 The use of hyperlinks in electronic documents relies on conventional user interface 

38 techniques — keying in and/or point and click — with a networked computing device. Conventional 

39 user interface techniques require the user to divert attention away from the hardcopy document (e.g. 
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1 a book) which he or she has been reading and require the user (a) to manually enter information 

2 (e.g. a WWW URL) needed to retrieve the related information and (b) if the user does not know 

3 beforehand and has not been informed in the hardcopy document where to access the related 

4 information, to perform some search in order to find the related information. 

5 Conventional techniques that rely on encoded information on a document have heretofore 

6 been quite limited, typically providing only for a limited set of information of a limited type, such as 

7 printed documents, to be retrieved and viewed. Also, some such techniques require the user to 

8 stop reading the document, to mark it in an appropriate way, and then to feed it manually into a fax 

9 machine or scanner coupled to a networked computing device, in order for the desired function to 

10 be performed. 

11 The techniques described by Sodergard et al. and Robinson et al. may lead to advances, 

12 but were developed with other objectives in mind and are not well-suited to efficiently obtaining an 

13 automatic action appropriate to a specific physical object. The Sodergard technique relies on OCR 

14 of URLs, but OCR is not reliable and a URL can appear in a hardcopy document even though the 

15 URL does not produce an action related the document. The Robinson technique requires a 

16 complex sequence in which a Web page is retrieved, then printed, then pointed to on the 

17 DigitalDesk. In general, these and other conventional techniques do not provide automatic actions 

18 appropriate to physical objects through a network in a non-disruptive streamlined manner. 

19 The invention provides techniques that alleviate these problems. The techniques employ 

20 action/medium identifiers encoded in machine-readable markings on marking media such as sheets 

21 or stickers of paper or documents. Each action/medium identifier identifies an action. The 

22 action/medium identifier can be used to obtain an action identifier that can be provided through a 

23 network to an action device to produce the action. The action device provides the identified action 

24 automatically in response to the action identifier. The action/medium identifier also identifies the 

25 marking medium. Because the action/medium identifier identifies both the marking medium and the 

26 appropriate automatic action, the marking medium can be used to obtain the appropriate automatic 

27 action in a non-disruptive streamlined manner. The user can obtain the automatic action in a way 

28 that does not disturb normal reading activity and does not disturb document appearance. 

29 Some of the techniques provide articles of manufacture for obtaining automatic actions 

30 through a network using processing circuitry for connecting to the network and detection circuitry for 

31 providing input signals to the processing circuitry. An article can include an area of a marking 

32 medium and machine-readable markings within the area. The machine-readable markings can 

33 encode an action/medium identifier as described above. The machine-readable markings can be 

34 decodabfe to obtain the action/medium identifier by the processing circuitry using input signals that 

35 define machine-readable markings in the area of the marking medium. 

36 The action/medium identifier can be implemented to include a globally unique identifier of 

37 the marking medium area, such as a globally unique page or sticker identifier. The detection device 

38 can be an image input device and the machine-readable markings can be invisible or can be 

39 visually nonobstructive markings. 
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1 If the area of the marking medium is a page, the action/medium identifier can include a 

2 page identifier, and the action device can include digital data defining a counterpart image of the 

3 page. The counterpart image can be isomorphic with the page. The action/medium identifier can 

4 also include a location identifier that identifies a location of a zone within the page, and the identified 

5 action can relate to the zone. The machine-readable markings that encode the page identifier can 

6 be in a different section of the zone than the markings that encode the location identifier. The zone 

7 can also include an orientation marking indicating orientation of the page. 

8 If the area of the marking medium is part of a hardcopy document, the action/medium 

9 identifier can include a document identifier that identifies the hardcopy document. The 

10 action/medium identifier can also include the action identifier, though the machine-readable 

11 markings encoding the document identifier can be at a different position than the markings that 

12 encode the action identifier. 

13 If the area of the marking medium is a sticker, the action/medium identifier can include a 

14 sticker identifier that identifies the sticker. 

15 In each case, the action/medium identifier can also include an access control code derived 

16 from the medium identifier using a secret function, to prevent certain types of fraudulent uses of 

17 identifiers. 

18 Other techniques provide methods of providing automatic actions through a network. A 

19 method can receive input signals from a detection device as described above. The method can use 

20 the input signals, decoding the machine-readable markings to obtain the action/medium identifier, 

21 can then use the action/medium identifier to obtain the action identifier, and can provide the action 

22 identifier through the network to the action device. 

23 If the area of the marking medium is a page, the method can be implemented with an action 

24 device that includes digital data defining a counterpart image of the page as described above. 

25 Second input signals defining machine-readable markings in a second area, encoding a second 

26 action/medium identifier, can be received and decoded. The second action/medium identifier can 

27 identify the marking medium and a location within it. The second area can also include human- 

28 readable markings identifying a link to a network address, and the action identified by the second 

29 action/medium identifier can follow the link. 

30 The method can also be implemented to provide the action/medium identifier to a machine, 

31 such as a global or network- wide router, to obtain network addresses. One network address can be 

32 that of the action device, the other that of a peripheral device that is also connected to the network. 

33 The method can use the action device's address to provide the action identifier and the peripheral 

34 device's address to the action device. The action device can respond by performing the action to 

35 obtain output data and can then use the peripheral device's address to provide the output data to 

36 the peripheral. 

37 The method can also be implemented to determine whether the action identifier meets a 

38 valid request criterion before it provides the action. If the action identifier includes both a medium 
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1 identifier and an access control code derived from the medium identifier using a secret code, the 

2 action device can determine whether the access control code was derived using the secret code. 

3 Yet other techniques provide systems for providing automatic actions through a network. A 

4 system can include detection circuitry for providing input signals as described above, and 

5 processing circuitry for receiving input signals. The processing circuitry can use the input signals to 

6 decode the machine- readable markings to obtain the action/medium identifier, then use the 

7 action/medium identifier to obtain the action identifier. The processing circuitry can then provide the 

8 action identifier through the network to the action device, which provides the action automatically. 

9 The system can be implemented with a pointer that includes the detection circuitry. The 

10 pointer can also include a user input device for providing user input signals, in response to which 

11 input signals are provided. The detection circuitry can be image input circuitry, the pointer can also 

12 include a marking element, and the processing circuitry can obtain, for a series of images, time data 

13 indicating the times at which the images occurred. The pointer can be a handheld device. 

14 The techniques according to the invention are advantageous because they can be used to 

15 provide an automatic action appropriate to a physical object, such as a hardcopy document or an 

16 object to which a sticker is attached, in a non-disruptive streamlined manner. 

17 In addition, the techniques according to the invention allow a rich variety of 

18 implementations. For example, in one implementation, the action/medium identifier can be encoded 

19 in invisible or visually nonobstructive machine-readable markings that are uniformly distributed on 

20 the marking medium; as a result, the detection circuitry can be in a pointer that can recover the 

21 action/medium identifier from any position on the page, and it is not necessary to provide the 

22 identifier in a text region, as would be the case for an action identifier that is a URL. Some 

23 implementations are advantageous because a handheld detection device can be used, increasing 

24 interactivity. In comparison with conventional techniques that rely on URLs, some implementations 

25 of the invention advantageously provide action/medium identifiers of sufficient density that they can 

26 identify any of an enormous number of actions and can allow worldwide or global access to remote 

27 information; URLs, on the other hand, require different words for different actions, which tends to 

28 limit the number of actions that can be addressed. Some implementations advantageously provide 

29 action/medium identifiers that can integrate in one encoded value a medium identifier, an action 

30 identifier, and an access control code. And some implementations allow a sheet of paper to act as 

31 a remote extension, analogous to an input peripheral, to a digital document repository, such as a 

32 collection of documents stored on a server connected to a network; the repository can be distributed 

33 as a result of the sheets acting as its extensions. 

34 The techniques according to the invention could thus be implemented to provide worldwide 

35 hyperlinking of paper documents to electronic data, within a virtually unlimited address space, and in 

36 a way that produces little or no distortion of usual document appearances and minimal disturbance 

37 of normal reading activity. 

38 The following description, the drawings, and the claims further set forth these and other 

39 aspects, objects, features, and advantages of the invention. 



5 



"Express Mail" No. EL214263395US 



Patent Application 
Attorney Docket No. R/98003C 



1 Brief Description of the Drawings 

2 Figure 1 illustrates schematically how a user may access electronic documents by 

3 interacting with hardcopy documents. 

4 Figure 2 shows schematically the routing of various data, including page identification codes 

5 and page location codes, over a network. 

6 Figure 3 shows schematically a sample of zones, and a disposition of machine-readable 

7 markings, on a coded substrate. 

8 Figure 4 illustrates components of a document printed on a coded substrate. 

9 Figures 5A and 5B show how digital data can be encoded in markings printed in a zone of 

10 the type illustrated in Fig. 4; 

11 Figure 6A shows schematically another disposition of zones or cells on a page and Figure 

12 6B shows schematically an exemplary zone or cell on the page. 

13 Figure 7 illustrates another exemplary zone in which machine-readable markings encode 

14 data. 

15 Figure 8 is a schematic illustration of internal components of a detection device or pointer. 

16 Figure 9 shows schematically a configuration for passing page identification codes and/or 

17 page location codes from a pointer as in Fig. 8 to a network computer. 

18 Figure 10 illustrates schematically an image containing coded zones that could be captured 

19 using an image input device or pointer as in Fig. 8. 

20 Figure 1 1 schematically illustrates features of an image input device or pointer. 

21 Figure 12 is a flow chart showing acts that can be performed in using identifiers to obtain 

22 network addresses. 

23 Figure 13 is a flow chart showing acts that can be performed in providing an action in 

24 response to identifiers. 

25 Figures 14 and 15 schematically illustrate isomorphism between physical (hardcopy) and 

26 electronic documents. 

27 Detailed Description 

28 The following definitions are helpful in understanding the broad scope of the invention, and 

29 the terms defined below have the indicated meanings throughout this application, including the 

30 claims. 

31 A "processor" or "processing circuitry" is a component that responds to input signals by 

32 performing processing operations on data and by providing output signals. The input signals may, 

33 for example, include instructions, although not all processors receive instructions. The input signals 

34 to a processor may include input data for the processor's operations. The output signals similarly 

35 may include output data resulting from the processor's operations. A processor or processing 

36 circuitry may include one or more central processing units or other processing components. 

37 A processor or processing circuitry performs an operation or a function "automatically" when 

38 it performs the operation or function independent of concurrent human intervention or control. 
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1 Any two components are "connected" when there is a combination of circuitry that can 

2 transfer signals from one of the components to the other. For example, two components are 

3 "connected" by any combination of connections between them that permits transfer of signals from 

4 one of the components to the other. 

5 An operation or event "transfers" an item of data from a first component to a second if the 

6 result of the operation or event is that an item of data in the second component is the same as an 

7 item of data that was in the first component prior to the operation or event. The first component 

8 "provides" the data, and the second component "receives" or "obtains" the data. 

9 A "network" is a combination of circuitry through which a connection for transfer of data can 

10 be established between machines. The circuitry in a network need not all be connected by 

11 transmissive elements such as wires-a network can include wireless links. Various types of 

12 machines can be connected to a network, and various terms can be used to refer to such machines; 

13 for example, the term "router" can be applied to a machine that responds to items of data by 

14 providing network addresses; the term "server" can be applied to a machine that responds to items 

15 of data by providing a service. 

16 An "action that can be produced" through a network is any operation that can be produced 

17 by providing signals through the network. An "automatic action" is an action that is automatically 

18 produced in response to appropriate signals. 

19 A processor "accesses" an item of data in memory by any operation that retrieves or 

20 modifies the item or information within the item, such as by reading or writing a location in memory 

21 that includes the item. 

22 A processor or other component of circuitry "uses" an item of data in performing an 

23 operation when the result of the operation depends on the value of the item. For example, the 

24 operation could use the item to access another item of data. 

25 An item of data "indicates" a thing, event, or characteristic when the item has a value that 

26 depends on the existence or occurrence of the thing, event, or characteristic or on a measure of the 

27 thing, event, or characteristic. An item of data "indicates" another value when the item's value is 

28 equal to or depends on the other value. 

29 An item of data "identifies" or "is an identifier of" one of a set of identifiable items if the item 

30 of data is one of a set of items of data, each of which can be mapped to at most one of the 

31 identifiable items. For example, an "action/medium identifier" is an item of data with a value that can 

32 be mapped to at most one of a set of actions and one of a set of marking media. 

33 An identifier is a "globally unique identifier" if it maps to at most one thing in the world. An 

34 identifier is a "network-wide identifier" if it maps to at most one entity available through a network. 

35 A first identifier "includes" a second identifier if the second identifier can be obtained from 

36 the first identifier. For example, the second identifier may be the same as the first identifier, it may 

37 be a part of the first identifier, or it may obtainable by decoding the first identifier or using the first 

38 identifier to access a table or other data structure to obtain the second identifier. 
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1 The term "image" is used herein to mean a pattern of light, whether within or outside the 

2 visible range. 

3 Signals "define" an image when the signals include sufficient information to produce the 

4 image. "Image input circuitry" is circuitry for obtaining data defining images as input. 

5 A "marking medium" is a physical medium on which a pattern of marks can be produced, 

6 such as by marking or printing actions. Common examples of marking media include sheets of 

7 paper and plastic. Where a collection of pieces of marking media such as a set of sheets are 

8 attached or otherwise associated to form a unit, the entire collection can be referred to as a marking 

9 media. As used herein, "marking" or "mark" includes both visible and invisible marks, and includes 

10 a single mark and also plural marks that together form a pattern of marks. 

11 Markings that encode items of data are "machine-readable" if techniques are available for 

12 automatically obtaining the items of data from signals that include information about the markings. 

13 Markings that encode items of data are "human-readable" if humans can perceive the markings and 

14 extract the items of data. 

15 A marking or collection of markings, even if visible, is "visually nonobstructive" if it does not 

16 obstruct a human's visual perception of other visible markings printed in the same area of marking 

17 medium. 

18 A "detection device" or "detection circuitry" is a component that detects markings and 

19 provides signals that include information about the markings. 

20 A "physical page" or "page" is a marking medium that includes only one sheet; a card, a 

21 poster, and an ordinary sheet are all pages within this meaning. A "hardcopy document" or 

22 "document" is a collection of one or more pages, with at least one page including markings 

23 indicating information. A "sticker" is a page that can be made to adhere to a surface. 

24 An "electronic document" is intended to cover not only electronically stored files which can 

25 be presented to the user visibly (e.g. via printing or electronic display of a page or document), but 

26 also to audio and audio-visual documents and any other records of information that can be stored 

27 and retrieved electronically, or computed or produced at will, and presented by appropriate means to 

28 the user. 

29 A "digital page" is an electronic document that is a counterpart of a physical page. A digital 

30 page and a physical page can have one or more types of "isomorphism", meaning ways in which 

31 they have the same form. For example, it may be possible to recover the exact layout of the 

32 physical page from the digital page because the position of any mark on the physical page is 

33 indicated, implicitly or explicitly, in the digital page. Due to isomorphism, a position in the physical 

34 page, such as a position indicated by a pointer, can also have meaning in relation to the digital page. 

35 Similarly, a "digital document" is an electronic document that is a counterpart of a hardcopy 

36 document, and the two may have isomorphism as described above. 

37 An activity is performed "on" a marking medium when the activity is performed in close 

38 proximity to the marking medium and the activity refers to the marking medium. Examples would 



8 



"Express Mail" No. EL214263395US 



Patent Application 
Attorney Docket No. R/98003C 



1 include making marks on the marking medium, scanning the marking medium, or pointing to a part 

2 of the marking medium. 

3 A "valid request criterion" is a criterion that can be applied to determine whether a request 

4 for an action is valid. 

5 1 . Introductory Overview 

6 1.1 Network Interactions 

7 Figs. 1 and 2 provide a useful introduction to network interactions available in some 

8 implementations of the invention. 

9 Figure 1 illustrates schematically a scenario in which a user may access electronic 

10 documents, and cause actions to be performed at a local or remote device connected to a network, 

11 by interacting with hardcopy documents. 

12 The user may manipulate (e.g. position or move with a wiping action) a detection device, 

13 such as a camera in pointer 502, an implementation of which is described in detail below, to capture 

14 local images of portions of hardcopy document 2 and provide input signals defining the captured 

15 images. Data derived from the captured images, such as action/medium identifiers, can be 

16 provided to a network (e.g. via some local device (not shown) provided with network 

17 communications facilities). Several examples of action/medium identifiers are discussed in further 

18 detail below. Document 2 can be, or can be formed from, a special type of marking medium in 

19 which a substrate, such as a sheet or sticker of paper or a document, bears visible or invisible 

20 markings that encode an identifier of the substrate and, in some cases, can also encode locations 

21 or zones within the substrate, as discussed in further detail below with reference to Figs. 3 to 7. The 

22 markings can, for example, be Xerox DataGlyphs. Such substrates are referred to herein as "coded 

23 substrates". In some of the examples described below, visible or invisible machine-readable 

24 markings encode a page identification code and, optionally, a page location code identifying a 

25 position of the markings relative to the page. 

26 Using this scenario, an action can be performed through the network, such as presentation 

27 to the user via a local device or peripheral device such as display 4 of output from a digital page 

28 related to document 2, or operation upon information related to or present in document 2. 

29 Figure 2 shows schematically the routing of various items of data, including page 

30 identification codes and page location codes, over a network between several components 

31 connected to the network. 

32 The user can click a button (not shown) on pointer 502, providing a signal similar to a 

33 mouse click, while the camera on the pointer is capturing an image of machine-readable markings 

34 either at a certain location on a physical page of document 2, or on a small sticker 2' on display 4. 

35 Sticker 2 l can be formed from a coded substrate. 

36 Processing circuitry receiving input signals from a camera in pointer 502 can decode the 

37 machine-readable markings to obtain two pieces of data — a page-identifier (pid) or sticker identifier 

38 (pid 1 ) and a location code (loc or loc') indicating localisation on the page or sticker. These decoding 

39 actions are shown as dotted lines extending from document 2 and sticker 2 1 to pointer 502. 
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1 The page-identifier pid (or sticker identifier pid'), with or without a location code, can be an 

2 example of an action/medium identifier. It can, for example, be a number that both identifies the 

3 page or sticker from which it is decoded and also identifies an action that can be produced through 

4 the network. In Fig. 2, the action is produced by using the pid (or pid 1 ) to obtain a network address, 

5 and by then providing the ioc as an action identifier. The network address can be obtained by 

6 sending the pid (or pid') to router 802, whose network address can be stored by pointer 502 or 

7 associated circuitry. Router 802 can use the pid (or pid') to obtain and return two pieces of 

8 information-a network address and a type which can illustratively be either "peripheral" for pid 1 or 

9 "digital-page" for pid. The association of pid (or pid 1 ) with its address and type can then be kept by 

10 pointer 502 or associated circuitry in a cache for possible future use. 

11 When the type is "peripheral", router 802 provides the network address of an associated 

12 peripheral device, such as the network address of display 4. Pointer 502 or associated circuitry can 

13 save the network address of display 4 in a peripheral-address store. In the implementation shown 

14 in Fig. 2, the location code Ioc' is not used when the type is "peripheral". 

15 When the type is "digital-page", router 802 provides the network address of digital page 6, 

16 which can be a Web page counterpart of document 2 that is stored on a server connected to the 

17 network. Pointer 502 or associated circuitry can (1) save the network address of digital page 6 in a 

18 digital-page-address store, and (2) send location code Ioc, serving as an action identifier, and the 

19 network address of display 4 from the peripheral-address store to the digital page's network 

20 address. 

21 The above technique is discussed further below with reference to Figure 12. 

22 Digital page 6 may be a digital representation (e.g. a Web page) of the printed, human- 

23 readable information in document 2, a representation that can be displayed or printed. Digital page 

24 6 may alternatively be a digital representation of some other document or information related to, or 

25 derivable from, document 2. 

26 When digital page 6, which can be located on a conventional Web server or other server or 

27 machine on the network (not shown), receives data from the pointer 502, an executable on the 

28 server on which the digital page resides can first be executed to decode the location code Ioc and to 

29 perform the action it identifies. The action typically produces output, and the same or another 

30 executable on the server on which the digital page resides can be executed to send the output to 

31 the peripheral address. The digital page can then wait to receive another Ioc and another peripheral 

32 address. The manner in which the digital page responds to action identifiers is discussed further 

33 below with reference to Figure 1 3. 

34 1.2 Further Network Considerations 

35 1.2,1 Routing 

36 Addresses can be interpreted globally (similar to Internet addresses) or interpreted pre- 

37 emptively by a local computer. 

38 The process of using or interpreting an action/medium identifier to obtain a network address 

39 is referred to herein as "routing". 
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1 Global or network-wide routing can be provided by an entity producing coded substrates 

2 ("coded substrate supplier). The coded substrate supplier can associate a default Web page, at 

3 some centralized site, with each page-identifier it produces. The buyer of a coded substrate page 

4 ipso facto becomes the owner of the centralized Web page associated with the page-identifier of the 

5 coded substrate page. The buyer may have only limited rights to use this page, due to memory 

6 usage considerations; for instance, the buyer may have only the right to store in it the net address of 

7 his own Web page, which can contain a digital page or other detailed data relating to information 

8 printed on the page. The centralized Web page provided by coded substrate supplier can act only 

9 as a go-between between the page-identifier and its associated digital page, and the machine on 

10 which the centralized Web page resides therefore acts as a router. With this approach, from 

11 anywhere in the world, the page-identifier can be routed to its corresponding centralized Web page 

12 and thence to its digital page through a generic routing scheme. 

13 This approach could, of course, be optimized in ways that do not require, for example, the 

14 owner of an address book living in Paris to route each page-identifier first through the headquarters 

15 of the coded substrate supplier in Palo Alto then back to his home computer in Paris. 

16 Routing could also be improved by caching a network path associated with a page-identifier 

17 for access efficiency. 

18 Various routing techniques that may be employed are described in greater detail in 

19 copending, coassigned U.S. Patent Application 09/CCC,CCC (Atty. Docket No. R/98005), entitled 

20 "Using Identifiers to Obtain Network Addresses, incorporated herein by reference. 

21 1.2.2 Wireless Communication, Ubiquitous Peripherals 

22 Other functions that the communication infrastructure may provide include: 

23 • allowing pointer 502 or associated circuitry to send a page-identifier to a predefined address 

24 router, which is able to determine the Internet address of the digital page corresponding to this 

25 page-identifier, 

26 • receiving (and possibly caching for future use) a digital page's Internet address, 

27 ♦ sending an encrypted code or other code indicating access rights to the digital page's 

28 Internet address, 

29 * sending location codes to the digital page's Internet address, 

30 ♦ sending to the digital page's Internet address the Internet address (or phone number, etc.) 

31 of a peripheral to which output should be sent, and 

32 * sending to the peripheral the output produced by a process associated with the digital page 

33 in response to a location code. 

34 2. Encoding Action/Medium Identifiers 

35 A wide variety of techniques could be used to encode and decode action/medium identifiers 

36 into and from machine-readable markings. 

37 Current or shortly available printing technology, conservatively estimated, permits to print on 

38 the order of 1000 bits of reliable information per square centimeter. For example, DataGiyph™ 

39 technology developed by Xerox Corporation already permits encoding on the order of 500 bits/cm 2 , 
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1 including a robust error correction scheme, assuming printing and scanning at 400 dpi. Considering 

2 the special constraints placed on DataGlyph™, which are required to be visually nonobstructive to 

3 humans, constraints that would not be necessary for invisible marks, and also keeping in mind that 

4 the production of coded substrate will be a process carried out by specialized machines, it seems 

5 reasonable to assume 1000 bits/cm 2 is achievable. 

6 Under address protocols currently being devised for the Internet, 128 bits theoretically 

7 permits addressing of 3.4 x 10 38 items, or 6.7 x 10 23 items per square meter of the surface of the 

8 Earth. In practice the assignment and routing of addresses requires the creation of hierarchies that 

9 reduce the efficiency of the usage of the address space. Therefore, these figures should be 

10 lowered somewhat, with a pessimistic estimate being 8 x 10 17 nodes (1564 addresses per square 

11 meter of the Earth), and an optimistic one being 2 x 10 33 nodes (4 x 10 18 addresses per square 

12 meter of the Earth). 

13 Each cm 2 zone or cell of machine-readable markings can hold, for example, 256 bits of 

14 reliable information. Of these, 128 bits can hold the page-identifier (which is thus redundantly 

15 repeated on each cell of a page), and 32 bits can hold the cell localization (or ceil address or 

16 location code) on the page. 32 bit location codes correspond to a maximum number of 65536 cells 

17 on a page, which in turn corresponds to a 123cm x 123cm maximum size page. 224 bits are left for 

18 other information, such as a page-id-code that can be used for authentication or data private to the 

19 publisher. 

20 Detection circuitry could be in a pointer that has the shape of a pen, as illustrated by pointer 

21 502 in Fig. 1. When the pointer's tip is positioned anywhere in a given cell, the detection circuitry 

22 can, for example, provide input signals defining an image showing the whole cell. By decoding 

23 machine-readable markings in the image to obtain the cell address, processing circuitry in or 

24 associated with the pointer can obtain the position of the cell on the page. By detecting the cell 

25 border, the processing circuitry can also determine the precise position of the pointer inside the ceil. 

26 Thus the processing circuitry can localize the pointer precisely relative to the whole page. The 

27 processing circuitry can also decode the page-identifier and the page-id-code and can then provde 

28 these two items of data, along with a location code, on the network for interpretation via external 

29 components. 

30 Various other techniques could be employed for encoding and decoding action/medium 

31 identifiers. 

32 2.1 Machine-Readable Markings 

33 A coded substrate supplier could use various techniques to produce machine-readable 

34 markings such as those described above. The coded substrate could, for example, include invisible 

35 markings on paper, which could be nonetheless be detected by the detection circuitry in a pointer 

36 device. 

37 For example, ultra-violet (UV) ink could be used to print invisible markings on white paper. 

38 Such an ink is transparent to the human eye, yet an area of paper covered by it reflect UV light 

39 differently from an uncovered area. A coded substrate supplier could produce sheets of paper in 
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1 different formats for different uses by the publishing industry. Each sheet can be processed through 

2 a specialized printing procedure which (1) assigns a fresh page-identifier (and possibly page-id- 

3 code) to the sheet, and (2) prints in UV ink machine-readable markings encoding the page-identifier 

4 (and possibly page-id-code) on the surface of the sheet. 

5 A publisher can buy these apparently uniformly white sheets and can print visible markings 

6 on them using standard ink. 

7 An end user can position a pointer device with detection circuitry on a printed page of the 

8 coded substrate. A radiation source on the pointer can emit UV light or the UV component of 

9 natural light can be used. The detection circuitry can detect the reflected UV light from the UV-ink 

10 layer. This light, apart from being invisible to the human eye, is similar to conventional visible light 

11 and standard techniques for optical scanning can be applied to obtain an input signal defining the 

12 UV light image. 

13 In certain cases it may be necessary to use different inks to avoid problems, such as 

14 conventional ink layers blocking transmission of UV light. However, in standard trichromatic 

15 printing, three layers of coloured inks (yellow, magenta, and cyan) are superposed to produce all 

16 visible colours. Each layer filters one part of the light spectrum independently of the others. That is, 

17 to a reasonable approximation, the magenta layer does not block light that would be absorbed by 

18 the cyan and yellow layers, etc. Therefore, for most trichromatic printing, UV light should not be 

19 blocked. 

20 Alternatives to UV ink include infrared ink, phosphorescent ink, or some type of magnetic 

21 ink, which has been proposed in the past for bar-code type scanning. It may also be possible to 

22 provide machine-readable markings from which an input signal can be obtained by non-optical 

23 means. It is beneficial, however, if the machine-readable markings are durable, do not interfere with 

24 visible printed marks, and permit recovery of page-identifiers, page-id-codes, and location codes. 

25 Other alternatives include visible inks printed in visually nonobstructive patterns, such as 

26 DataGlyphs, which can provide robust encoding and decoding of data at a relatively high density. 

27 Figure 3 illustrates a sample of zones or cells, and the disposition of machine-readable 

28 markings, on an coded substrate. Fig. 3 shows that each zone or cell 202 can include markings as 

29 follows: 

30 cell border 204, which must be distinguishable from cell contents, and which 

31 surrounds the interior of cell 202; 

32 some indication of the orientation of the page, such as orientation marker 206, 

33 which can be within the interior of cell 202; 

34 an encoded representation of a page-identifier, i.e. an item of data whose value 

35 uniquely identifes the page, within cell border 204, such as a first set of markings 208; and 

36 an encoded representation of localization information, such as a location code 

37 uniquely defining the position of cell 202 within the page, within cell border 204, such as a second 

38 set of markings 21 0, illustratively smaller in size than first set 208. 



13 



Express Mail" No. EL214263395US 



Patent Application 
Attorney Docket No. R/98003C 



1 Each zone or cell 202 thus includes border 204 and orientation marker 206. A first set of 

2 markings 208 over part of the interior of cell 202 provides an encoded representation of a page- 

3 identifier, while a second set of markings 210 over a smaller part of the interior of cell 202 provides 

4 an encoded representation of a location code (page-loc) ~ uniquely defining the position of cell 202 

5 within the page. The orientation of the page is given by mark 206 in one comer of cell 202, making it 

6 asymmetric in both dimensions. The cells can be tiled to cover the entire page. 

7 In Fig. 3, the page-identifier and location code are shown schematically; detail is given in 

8 Fig. 5, discussed below. 

9 Figure 4 illustrates components of a document printed on a coded substrate. Printed 

10 document 102 comprises layer 104 of printed visible (human-readable) information, i.e. document 

11 content, printed on coded substrate 106, illustratively a segment of a map though layer 104 could 

12 include text, photographic images, or any other human-readable information. The coded substrate 

13 106 in turn comprises a layer 108 of visible or invisible machine-readable markingsprinted on a 

14 sheet medium 1 1 0 (e.g. paper). 

15 Figures 5A and 5B show how digital data can be encoded in a single zone or cell like those 

16 illustrated in Fig. 2. Fig. 5A shows the unencoded binary data with each binary value or bit in 

17 approximately the position of its encoded representation, i.e. 47 bits of page-identifier in upper 

18 section 302 (the bit stream wraps at cell border 204), and 16 bits of page localization data (loc) in 

19 the lower section 304. The page-identifier code denotes 108429159095492 (decimal) = 

20 629DA182DCC4 (hexadecimal) 

21 110001010011101101000011000001011011100110001000001000000010101 (binary, to make the 

22 wrapping explicit). Within the border of a cell, the page-id is represented as an ordinal number, in 

23 binary (Fig. 5A). The cell can be 8 by 8 symbols (or bits) wide, with one symbol's worth of space 

24 taken up by the orientation mark. In the 16 bit loc code in section 304, there are 8 bits for the X 

25 coordinate and 8 bits for the Y coordinate. Thus, for the cell (zone) shown, its position is 16,21 on 

26 the substrate. Localization identifies a cell (zone) on the page, not a point on the page. 

27 With this encoding, a complete cell must be within the region of the page from whose image 

28 a page-identifier and location code are to be obtained. Less than a complete cell does not contain 

29 sufficient information. 

30 The addition of error correction information is desirable. This can be done with standard 

31 techniques and is not shown; the actual encoding after the addition of error correction information 

32 will be different from what is shown, and will require more bits (and thus more space). 

33 The number of possible pages and the number of possible cells on a page are a function of 

34 the density of the encoding. Using the encoding shown, 2 47 pages can be identified, with up to 64 

35 by 64 cells on each. 

36 Fig. 5B shows the same data as in Fig. 5A, but represented by DataGlyph markings. 

37 Encoding using DataGiyphs and the retrieval of data therefrom is discussed further in US-A- 

38 5,486,686 and EP-A-469864, both incorporated herein by reference. Here, a first set of glyphs 

39 (markings) in upper section 402 encodes a page-identifier and a second set in lower section 404 
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1 encodes a location code (ioc). Each glyph encodes the bit at approximately the same position in 

2 Fig. 5A, so that each glyph extending from upper left to lower right encodes a "1" and each glyph 

3 extending from lower left to upper right encodes a "0". DataGlyphs offer advantages in robust 

4 decoding. 

5 A page-identifier, location codes, and any other data can be encoded on the coded 

6 substrate in any other suitable manner. For example, the encoding may be as described in any of: 

7 U.S. patents 4,786,940; 5,245,165; 5,278,400; 5,315,098; 5,449,895; 5,521,372; 5,537,223 ; 

8 5,572,010 and 5,576,532. 

9 The inks used to encode data may be visible or invisible, and may be detected by visible 

10 optical, IR, UV, magnetic, or any other suitable means. For example, suitable inks for the machine- 

11 readable markings may be as described in any of U.S. patents 5,075,186; 5,225,900; 5,301,044; 

12 5,145,518; 5,046,087; 5,275,647; 5,208,630; 5,202,265; 5,271,764; 5,256,193; 5,286,286; 

13 5,331,140; and 5,291,243. 

14 Figure 6A shows a disposition of zones on a page, suggesting how a location code could be 

15 based on successive divisions of regions of the page into quadrants rather than on X and Y 

16 coordinates as described above. Figure 6B shows an exemplary zone or cell, with a page-identifier 

17 encoded in the center of the cell in alphanumeric characters and with a location code encoded by a 

18 sequence of embedded concentric localization markings, each of which is a quasi-square with a 

19 corner missing, the orientation of the missing corner of each localization marking thus indicating one 

20 of four values corresponding to the four quadrants illustrated in Fig. 6A; by starting with the 

21 outermost localization marking in Fig. 6B, and by taking the indicated quadrant, the specified cell in 

22 Fig. 6A is reached. 

23 Figure 7 illustrates another exemplary zone in which machine-readable markings encode 

24 information. Here, a paper page is divided into 0.5 cm x 0.5 cm square cells. Each such cell can 

25 contain on the order of 256 bits of information. The illustrated cell 202 is identifiable by its distinctive 

26 border 204, consisting of uninterrupted horizontal and vertical lines of black pixels. A square of 

27 pixels, which is separated from the black border by a white area one pixel wide, holds the encoded 

28 information, illustratively 10 pixels x 10 pixels, but in reality it could have a higher resolution. 

29 Machine-readable markings could encode information in various other ways. For example, 

30 rather than encoding a page identifier with location codes over a large region of a page as 

31 suggested above, encoded location identifiers could occupy relatively small regions and could be 

32 positioned only in certain active positions or regions of the page, such as rectangular regions 

33 around words or icons printed on the page. This implementation is referred to herein as a "non- 
34 positional implementation". 

35 The encoded location identifiers in a non-positional implementation could, for example, be 

36 DataGlyph address space fragments of the type described in copending, coassigned U.S. Patent 

37 Application No. 08/772,158, entitled "Parallel Propagating Embedded Binary Sequences for 

38 Characterizing Objects in N-Dimensional Address Space", incorporated herein by reference. Other 

39 markings could be used, such as bar codes, icons, circled numbers in an OCR font, and so forth. 
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1 The robustness of the non-positionai implementation increases with the ease of recognition of the 

2 markings, while the document looks better as the intrusiveness of the marks decreases, and 

3 particularly as the marks remain recognizable when partially occluded, metrics on which DataGlyphs 

4 perform well. 

5 If a page or document includes no more than 1000 active positions or regions, 10 encoded 

6 bits would be sufficient to uniquely identify each active position or region. If printed in visible ink, the 

7 machine-readable markings could provide a visual cue reminding the user that a position or region 

8 is active, even though the markings are visually nonobstructive. 

9 For a multipage document, a non-positional implementation could also provide a unique 

10 document identifier that is encoded in machine-readable markings on each page of the document, 

11 making a page-identifier unnecessary. The document identifier could be encoded in each active 

12 position or region together with the position identifier, or it could be encoded separately in markings 

13 that could be at the bottom of the page. Alternatively, the document identifier could be encoded only 

14 once, such as on a cover page of the document. 

15 3. Detection and Processing 

16 As mentioned above, a detection device or detection circuitry can provide input signals that 

17 include information defining the machine-readable markings, and processing circuitry can use the 

18 input signals to obtain an action/medium identifier. The detection device or circuitry and the 

19 processing circuitry could take any appropriate form. Some examples are illustrated in Figs. 8-11. 

20 Other examples of detection devices that could be used are described in copending, coassigned 

21 U.S. Patent Applications 09/144,250, entitled "Methods and Apparatus for Camera Pen"; 

22 09/144,251, entitled "Glyph Address Carpet Methods and Apparatus for Providing Location 

23 Information in a Multidimensional Address Space"; and 09/233,882, entitled "Multi-level Selection 

24 Methods and Apparatus Using Context Identification for Embedded Data Graphical User 

25 Interfaces", all incorporated herein by reference. 

26 In Figs. 8-1 1 , the detection device is included in a pen-shaped pointer, that can be provided 

27 with a clicker or other user input device. The pointer could include a small transparent screen, 

28 rimmed with a code-reading device, with a cross hair in the middle; this may be a less costly 

29 implementation in the near term than a pen-shaped pointer, and it can encompass more of the 

30 surface of a page at one time thus allowing for more accurate recovery of position and/or the 

31 possibility of recovering the position with fewer coded areas on the page. 

32 Figure 8 is a schematic illustration of internal components of an image input device that is 

33 part of a pointer. The pointer is a complex object that may be a single physical device or may be 

34 several communicating devices. It can include: 

35 1 . A hand-held image-reading device, including a clicker or button; 

36 2. An image-decoding device; and 

37 3. A network communication device. 

38 The image-reading device receives light and produces image data. When the user clicks 

39 the button, the image data is made available to the image-decoding device. The image-decoding 
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1 device decodes the image, producing a <pid, loo pair which can be used by the network 

2 communication device as illustrated in Fig. 12. 

3 Figure 8 shows schematically a special purpose pointer. The illustrated pointer includes 

4 camera 802, frame grabber 804, memory 806, central processing unit (CPU) 808, a button (not 

5 shown, but can be similar to a conventional mouse button) for the user to click on a region of a page 

6 to activate frame grabber 804, memory 809 which includes read-only memory (ROM) storing image 

7 decoding software, network communication software, and control software, and network connection 

8 hardware 810. The functions of these components could be provided by any appropriate 

9 combination of software hardware. Memory 809 can also include registers or other appropriate 

10 storage for a digital-page network address, an output-peripheral network address, and a cache 

11 comprising a set of triples <pid, type, address>, where "type" distinguishes the pid of a digital page 

12 from that of a peripheral. 

13 The image-reading device can be implemented as a pen-camera or hand-held scanner, 

14 typically including CCD elements with hardware to turn optical input into image data. 

15 The pointer can also be implemented as a collection of devices that communicate with each 

16 other rather than as a single device. The most obvious separation points are mentioned in 1 , 2, and 

17 3 above, though 1 and 2 could be in the same physical device, or 2 and 3. 

18 The image-decoding device may also be implemented as a general purpose computer 

19 containing a frame grabber board and an image decoding software package. Likewise the network 

20 communication device can be implemented as a general purpose computer with a network 

21 connection or as a network computer. 

22 The function of frame grabber 804 is to capture a snapshot in memory from the image data 

23 that camera 802 continuously produces. If camera 802 is not integrated with or tethered to the 

24 image-decoding device, both devices must have wireless communication facilities to allow camera 

25 802 to transit image data to frame grabber 804. There are performance considerations in such an 

26 implementation, because the amount of image data necessary to define an image is large. The 

27 button click also must be communicated, as it resides on camera 802 for ergonomic reasons, but 

28 activates frame grabber 804. 

29 The snapshot can be input to the image-decoding device, whose output can be a <pid, loo 

30 pair. If the image-decoding device is not integrated with the network communication device, the 

31 <pid, loo pair can be transmitted to the network communication device. Again, each device can 

32 have communication facilities, such as a cord, or some form of wireless communication such as 

33 infra-red, or mobile telephony. 

34 An alternate implementation of a pointer is shown in Fig. 9. Fig. 9 shows a configuration for 

35 passing page identification codes and/or page location codes from image input circuitry as in pointer 

36 502 in Fig. 1 to processing circuitry that includes a network computer. Image capture device 506, 

37 illustratively a CCD or other appropriate camera, is coupled by wired or wireless (e.g. IR or RF) link 

38 to processing device 602 and in use provides image data defining capture images to processing 
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1 device 602. Image capture device 506 and processing device 602 may be combined into a single 

2 handheld unit or may be separate components. 

3 The operative elements of processing device 602 include frame grabber circuit 604, image 

4 decoding software 606, and CPU 608, which are known in the art. CPU 608 can execute image 

5 decoding software 606 and other conventional software to receive and decode an image from frame 

6 grabber circuit 604 and to provide decoded data to other components as described below. 

7 Processing device 602 can also include memory, accessible by CPU 608 for storage and retrieval of 

8 data. 

9 In use, processing device 602 extracts from the image data the encoded page-identifier and 

10 page-location data to obtain an item of data (<pid, loo) and communicates the item of data in a 

11 wired or wireless fashion to a local device illustratively network computer 610 linked to a network 

12 such as an intranet or the Internet in a known manner. Computer 610 can have its own unique 

13 network address, but need not have any information output device like a display screen or printer, 

14 because such a device is not necessary to perform the function of providing a connection between 

15 CPU 608 and the network. Processing device 602 and computer 610 could therefore both be within 

16 pointer 502, and could communicate and share resources through any appropriate circuitry. 

17 Computer 610 could, however, perform other functions, in which case it could have a user interface 

18 appropriate for those functions. 

19 Network computer 610 can have a cache and digital-page network address and output- 

20 peripheral network address storage. As network computer 610 need not have a screen, it need not 

21 be large. 

22 Figure 10 illustrates the capture of an image containing coded zones 202 using an image 

23 input device such as a camera in a pointer. Fig. 1 0 shows a small region of a page, with field of view 

24 of the camera (not shown) delimited by circle 702. The radius r of the camera's field of view must 

25 be larger than the diagonal of cells 202 in order that the field of view contain at least one cell 202. 

26 The center 704 of the camera's field of view (shown as the intersection of cross-hairs 706) will lie in 

27 exactly one cell, or, if it lies on cell border 204, one of the adjacent cells is chosen by convention as 

28 the cell that contains center 704. 

29 The location of center 704 relative to cell borders 204 identifies a location within a cell 202, 

30 which in combination with the orientation and the identification of cell 202 identifies the location on 

31 the page. 

32 Figure 1 1 schematically illustrates features of a pointer that may be used in implementing 

33 the invention. Pointer 502 comprises pointing device 504, which may have tip 505, and image 

34 capture device 506, which can be a camera or another type of image input circuitry. In use, image 

35 capture device 506 is able to capture images of an area A of a document 508, which can be 

36 implemented with a coded substrate as illustrated in Figs. 3-7. For the sake of illustration, the sizes 

37 of the components of pointer 502 are exaggerated, and pointing device 504 is shown parallel to 

38 image capture device 506 although they could be differently oriented - e.g. in practice, area A may 

39 be much closer to tip 505 of pointing device 504 than appears in Fig. 11 , so that a user may easily 
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1 determine from tip 505 whre area A is located. In certain embodiments, pointing device 504 may be 

2 omitted. In others, pointing device 504 may also be a pen or any other marking device suitable for 

3 making marks on a coded substrate that are visible to a user.lf pointing device 504 is a marking 

4 device, it functions as a combination writer-pointer. Image capture device 506 can monitor the 

5 position of tip 505 in real time, and marks can be recorded in digital form concurrently with their 

6 physical production. The allows capture of handwritten notes. If pointer movements are 

7 timestamped, the dynamicity of the strokes can be recovered to preserve the time and order of each 

8 annotation and its relations to other events, such as events in a video recording; related techniques 

9 are described in copending, coassigned U.S. Patent Application 09/BBB,BBB (Atty. Docket No. 

10 R/98004), entitled "Obtaining and Using Data Associating Annotating Activities With Portions of 

11 Recordings", incorporated herein by reference. The dynamicity of the strokes can also be used in 

12 signature authentication and handwriting recognition. Editor's marks handwritten on a draft 

13 typescript can be interpreted in real time to produce a corrected version. If the pointer is wireless, 

14 notes can be available online, allowing applications such as the transmission of handwritten faxes 

15 without a fax machine nearby. 

16 The document 508 may be a 'blank' coded substrate, or such a substrate having human- 

17 readable information printed thereon. 

18 A pointer as described above can have additional memory storing a unique pointer 

19 identifier, which the pointer can transmit along with an action/medium identifier such as a <pid, ioo 

20 pair. The pointer identifier can also be used to identify the pointer user. This technique makes it 

21 possible to provide a newspaper or magazine clipping service: When a user clicks on an article with 

22 the pointer, a hardcopy of the article is delivered to the user. The newspaper or magazine need not 

23 be tailored to the user in any way, but a subscribing user can provide a pointer identifier and a fax 

24 number. Then, when the user clicks or clips an article, the pointer identifier is looked up by the 

25 server in a table it maintains to obtain the fax number, which is used to fax the selected article. 

26 A pointer as described above can also have additional programmable memory whose 

27 contents it transmits along with an action/medium identifier such as a <pid, loo pair, for use in 

28 relation to a digital page. In this case, the user need not subscribe to the clipping service, or provide 

29 identification in advance. The programmable memory can include a fax number and a credit card 

30 number or other payment authorization code. When the user clicks or clips an article, these items 

31 are also transmited, so that the user pays for clippings as he or she goes. 

32 These techniques make it possible to tailor services to the user of the document, so that 

33 two users, performing the same actions with the same document, may obtain different results. 

34 Services can thus be tailored for ease of use, without a separate action to transmit one's fax 

35 number or to pay. 

36 4. Producing Actions 

37 Input signals that include information identifying machine-readable markings could be 

38 processed in a variety of ways to obtain action/medium identifier, to use an action/medium identifier 
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1 to obtain an action identifier, and to provide the action identifier on a network to produce the 

2 identified action. Figs. 1 2 and 1 3 illustrate one way these operations could be performed. 

3 Figure 12 is a flow chart illustrating acts that can be performed by processing circuitry, such 

4 as by CPU 608 or network computer 61 0 in Fig. 9 or by CPU 808 in Fig. 8. 

5 In the first box in Fig. 12, the processing circuitry initializes memory locations at which it can 

6 store two network addresses, an address for a peripheral device to which output can be sent and an 

7 address of a digital page. As illustrated, the processing circuitry then performs a series of iterations, 

8 each of which begins when a user clicks. 

9 In the second box, the user clicks, providing a signal that causes an image signal that 

10 includes information about machine-readable markings to be received. In the third box, the 

11 processing circuitry uses the image signal to obtain a decoded action/medium identifier in the form 

12 of a <pid, loo pair. In the fourth box, the processing circuitry determines whether the pid can be 

13 mapped to a network address using cached data from previous operations. If not, the processing 

14 circuitry provides the pid to a router in the fifth box and the router provides a network address and a 

15 type in response. In the sixth box, the processing circuitry stores the pid in its cache, together with 

16 the network address and type from the router. 

17 In the seventh box, the processing circuitry branches based on the type received from the 

18 router or retrieved from cache. If the type is peripheral, the processing circuitry follows the right 

19 branch in Fig. 12, and sets its peripheral address to the network address received from the router or 

20 retrieved from cache. If the type is digital page, the processing circuitry follows the right branch, first 

21 setting its digital page address to the network address and then sending the peripheral address and 

22 loc to the digital page address. 

23 Figure 13 is a flow chart of acts that can be performed by an action device at which a digital 

24 electronic document such as a digital page is stored. As illustrated, the action device performs a 

25 series of iterations, each of which begins with the action device waiting to receive a peripheral 

26 address and a loc from the network, as shown in the first box. 

27 When a peripheral address and a loc are received, the action device decodes the loc to 

28 obtain data indicating an action to be performed. The decoded data may, for example, be a handle 

29 for calling an executable to perform the action. The decoded data may include a file identifier for a 

30 digital page and an application for opening the digital page, such as Microsoft Word or PowerPoint. 

31 Then, the action device performs the action, obtaining output. In the fourth box, the action device 

32 provides the output to the peripheral address through the network, and the device at the peripheral 

33 address can display or otherwise respond to the output. 

34 The acts in Figs. 12 and 13 could be modified slightly for a non-positional implementation. 

35 Rather than providing a single click as in Fig. 12, the user can first click to provide an image signal 

36 including information about the document identifier at the bottom of a page or on a cover page, and 

37 then can click again to provide an image signal from one of the active regions with an action 

38 identifier, that is, a number identifying the region relative to the document. The processing circuitry 

39 can use the image signals to obtain a <doc, act> pair with the two identifiers. The processing 
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1 circuitry can provide the doc to the router to obtain a digital document's network address, and can 

2 then provide the peripheral address and the act to the digital document's address through the 

3 network. The action device can respond much the same as in Fig. 13, using the act rather than loc 

4 to obtain data indicating an action to be performed. 

5 As noted above, a non-positional implementation could alternatively encode the document 

6 identifier with the action identifier at each active position or region. If the action identifiers of 

7 different documents are in distinct ranges or sets of values, no further encoding is necessary 

8 because the action identifiers implicitly encode document identifiers. If a document identifier is 

9 encoded, implicitly or explicitly, with the action identifier at an active position or region, a single click 

10 could be used as shown in Fig. 12 to provide an image signal including information about both 

11 identifiers. 

12 5. Other Implementations 

13 5.1 Different Networks 

14 The above description is quite general, and could be implemented with various 

15 communication protocols and infrastructures, such as infrastructures including conventional wired or 

16 wireless communication between pointers, peripherals, and computers. 

17 Other implementations may couple coded substrates and peripherals in other ways. For 

18 example: 

19 • Shakespeare's play Othello, or any other such work, could be published as a 

20 book/CD/pointer/software package for use on a local network with a computer containing a CD 

21 player. If the CD player can interact with a monitor, the same work could be published as a 

22 book/CD/pointer package, where the software to control the CD player is built into the pointer. 

23 Signals from the pointer can be provided to the computer through the local network. 

24 • The pointer can contain a wireless telephone, and a coded substrate sales 

25 catalogue could allow a user to place an order by clicking on an item. The catalogue provider's 

26 phone number could be encoded in the substrate along with the page-id and pointer location 

27 information. Clicking on an item then initiates a call to the encoded number through a wireless 

28 network. The page-identifier and pointer location can be transmitted to the catalogue provider. 

29 These items of data can be used to identify both the item and the customer, allowing an order to be 

30 placed. 

31 5.2 Other Devices, Marking Media, and Communications 

32 5.2.1 Detection Devices 

33 A variety of implementations could employ different detection devices. Examples include: 

34 A detection device with a sensitive transparent screen overlaid on a coded 

35 substrate. A flexible, transparent screen made of plastic, rubber, or etc., could have, in one or more 

36 of its corners, a flat coded substrate code reading device. When the screen is touched with a 

37 finger, the screen is able to determine the location of the finger relative to the screen. Such a 

38 screen can for instance serve as a universal keyboard: It could be laid over a coded substrate on 

39 which a keyboard has been drawn. The user can types at a certain location on the screen, and the 
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1 screen can determine what the location is, not only relative to its own coordinates, but also, because 

2 of the information read at the corners, relative to the intrinsic paper coordinates. The infrastructure 

3 can then determine that this is the location of letter A, for example. An action identifier could be 

4 obtained and used to obtain an action as described above, such as presentation of the letter A on a 

5 display. 

6 Such transparent screens could be more practical than a pointer when several fingers act in 

7 parallel, as when typing on a keyboard. Some paper-based musical instruments could be 

8 implemented on this principle, with the actions being the generation of sounds. 

9 Technology may become available to provide a transparent screen with a display capacity. 

10 In this case, an action could be to overlay the printed marks on physical paper, seen by the 

11 transparent sheet, with dynamically displayed graphics, in a way sensitive to the location of the 

12 screen on the page. 

13 • Another type of detection device could allow fingers to touch the paper directly, 

14 while finger locations are recorded through a pressure sensitive clipboard or the like positioned 

15 under the paper, rather than over it. 

16 • Another type of detection device which would not require either a screen or a 

17 clipboard could be a glove with a finger tip holding a small image input device. 

18 • Other types of detection devices that provide printing capability are a bi-directional 

19 writer-pointer and a hand-held stamper-like printing machine. 

20 The writer-pointer illustrated in Fig. 1 1 could be modified to provide a bi-directional writer- 

21 pointer which receives external commands to deliver or not to deliver ink depending on its current 

22 position relative to the paper sheet. The user can cover an area by quick forward and backward 

23 moves of the pen; the positions at which ink has been deposited approximate a piece of graphics 

24 sent by the external site; in this way a limited local printing capability can be simulated. 

25 Another way to obtain a local printing capability is through a small handheld printing device, 

26 resembling a stamper. The user can place the device over areas of the paper sheet; for each such 

27 placement, the stamper is able to determine its precise position relative to the sheet coordinates, 

28 and it can also determine which parts of the area still remain to be covered by marks. By repeated 

29 rough placements of the device by the user, a piece of graphics can be printed with high precision at 

30 a specific location on the paper sheet. 

31 • Another detection device is a hand-held precision scanner-pen. 

32 The precise localization capability rendered possible by a coded substrate can be exploited 

33 for scanning purposes. Suppose the user has produced some drawing or note in the margin of a 

34 coded substrate. A small pen-like device can be provided both with a localization capability and with 

35 a scanning capability. By moving this device over the area of interest, if necessary through 

36 repeated moves, the user can perform a precise digitization of the whole drawing; this can be done 

37 even if the device has a small footstep (active scanning surface), because the images scanned 

38 during several moves will be easy to connect into a global picture. 
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1 The action in this case might include a process that pastes each scanned image in a page 

2 frame, positioned consistently with the location code. This technique would allow quick and 

3 accurate production over a network of a complete digital image of a region scanned in several 

4 independent wiping actions. 

5 5.2.2 Marking Media 

6 The above implementations employ marking media such as pages and documents, but a 

7 variety of other marking media could be employed. An example is: 

8 • Small round stickers, referred to as "confetti", could be made from a coded 

9 substrate. In particular, each sticker can have a unique sticker identifier. The stickers can be used 

10 to add multimedia links to a piece of conventional paper. The user can buy a few coded confetti. 

11 When perusing a conventional document, printed on standard paper, if the user wants to associate 

12 a certain action with a certain location on a given page of the document, such as to go to a certain 

13 Web site, to ring a certain phone number, to print a certain address on an envelope, the user can 

14 stick the confetti (which could be coloured or transparent) at the location of interest and associate, 

15 through his personal computer, or through a service provided by the confetti provider, the action 

16 with the confetti's sticker identifier. 

17 Coded confetti can also be seen as a transition technology that permits one to confer upon 

18 conventional documents some of the advantages of coded substrate documents. Obviously confetti 

19 could be stuck on any support, not only printed material. A confetti stuck to a machine part could 

20 have the effect of showing a description of this piece on a screen, when pointed to. 

21 Coded confetti could also be used to establish links between network connectted objects, 

22 such as peripheral devices. The user could stick a confetti to a peripheral, and the associated 

23 action of the sticker identifier could be to establish the peripheral as the current peripheral to which 

24 output is transmitted. The action could also include transmitting information about the peripheral's 

25 characteristics to permit another action at the digital page to be performed or optimized as 

26 appropriate to the peripheral. 

27 5.2.3 Communications 

28 A variety of communication techniques could be implemented using the above techniques, 

29 including the following: 

30 Communication could be provided between documents. Although there may be cases 

31 where every desirable capability of a coded substrate document is provided by its publisher 

32 independently of the existence of other coded substrate documents, this may not always be true. 

33 Communication can be established among coded substrate documents produced by different 

34 publishers, increasing overall document functionality. 

35 Interdocument communication could be provided by using the pointer to send its own 

36 identifier with every transmission, and the pointer's identifier could be used to establish 

37 communication. For example, the pointer identifier could be associated with a service that allows 

38 an application to store or retrieve data, so that one application providing a document could store 

39 data and another application providing another document could retrieve the data. The pointer 
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1 identifier could include the network address of the service, or a router could use the pointer identifier 

2 to obtain the network address. The router could be dedicated to pointer identifiers or could handle 

3 them in the same address space as page identifiers. 

4 Alternatively, the service could allow digital pages to register their own addresses, possibly 

5 with information about what sort of communication they can establish, and to retrieve such 

6 information registered by another digital page. For example, a digital page could register a remote 

7 procedure call which, when called, would establish a particular kind of communication, for retrieval 

8 and use by another digital page in making a call to the procedure. 

9 Interdocument communication could be provided by extending the pointer to incorporate an 

10 internal communication service as described above. The pointer identifier in this case would be the 

1 1 pointer's Internet address, and routing would be unnecessary. 

12 Interdocument communication could also be provided by extending the pointer to include a 

13 mode in which page identifiers are treated as data upon the user's request. A second button or the 

14 like could be clicked to have a page identifier stored in the pointer as data and transmitted on the 

15 next regular click along with the location code for the next regular click to the digital page associated 

16 with the page of the next regular click. The digital page could use the page identifier from the first 

17 click to establish a connection such as a socket between two digital page applications. In this case, 

18 the digital pages are in more of a master/slave relationship, because one receives the other's 

19 address. Also, the user must specify which clicks are treated in which way. 

20 The value of interdocument communication can be understood from two examples. 

21 • Computerless email: Ariane wants to send a message to her friend Irene. She 

22 writes the message, using a writer-pointer as inFig. 11, on NotePaper, a pad of coded substrates 

23 designed for sending notes. She then looks up Irene in her address book, itself a coded substrate 

24 document. She clicks the pointer on a Fax icon in Irene's listing in the address book. She then 

25 clicks on a Send icon on the sheet of NotePaper containing the message. The NotePaper 

26 application sends the image of that sheet of NotePaper to the address book application, which 

27 locates Irene's fax number in its database, and faxes the message to Irene. 

28 In another context, Ariane could write another message on a sheet of NotePaper, and an 

29 image of the message could be recorded and stored on NotePaper's digital page based on signals 

30 from the writer-pointer. When Ariane clicks on the Print icon on a page in her address book, that 

31 page's digital page process receives the pointer identifier, resolves it to a network address of the 

32 pointer's registry service, and sends information sufficent to establish a socket connection with the 

33 address book page's digital page process and a request for taxable data. When she then clicks on 

34 the Send icon on the NotePaper, the NotePaper's digital page process receives the pointer 

35 identifier, resolves it to a network address of the registry service, and retrieves the information sent 

36 to the registry by the address book process, then establishing the socket connection and 

37 transmitting the stored image of the message. The address book process receives the image and 

38 sends it to the fax number associated with the Fax icon's active region. Or the Send icon could be 

39 pressed before the Fax icon, and the NotePaper digital page process could send the image to a 
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1 data storage location associated with the pointer identifier, after which the Fax icon would cause the 

2 address book process to retrieve the image and fax it. 

3 The registry or storage could be in the pointer, making it unnecessary to resolve the pointer 

4 identifier to a network address. 

5 In another context, Ariane could write her message and store it as above, then click 

6 anywhere on the page using the second button. The page identifier of NotePaper is obtained and 

7 stored in the pointer. Ariane then clicks on the Fax icon of the address book page, and the page 

8 identifier of NotePaper as well as the location of the Fax icon in the address book are provided to 

9 the address book process. The action associated with the Fax icon receives the page identifier and 

10 uses it to establish a socket connection with NotePaper's digital page process. The socket 

11 connection can then be used to request the image, which the NotePaper digital page provides and 

12 the address book faxes as above. 

13 These examples show how any data associated with a coded substrate can be sent to any 

14 peripheral addressable by another coded substrate. Specifically, a handwritten message can be 

15 sent to someone's fax machine or computer without the sender interacting with a fax machine or 

16 computer. 

17 • Comprehension Aid: Ariane is at home reading an English novel in a coded 

18 substrate edition that does not provide translation. She decides to get a French gloss of a difficult 

19 paragraph. From her wallet she takes a utility card with a number of application icons, and clicks on 

20 English->French Glosser. She then clicks on a Send Region on the inside of the back cover of her 

21 novel, and subsequently on the beginning and end of the difficult paragraph. The novel's application 

22 sends the paragraph to the glosser application, which displays the paragraph with a French gloss on 

23 Ariane's television screen. 

24 6. Additional Functionalities 

25 6.1 Basics 

26 A digital page can be viewed as an object comprising functions and data that is constructed 

27 as the active counterpart to a physical page. As such, it implements any actions initiated by the 

28 user interaction with the physical page. The term "digital page process" refers to an executing 

29 instance of a digital page. 

30 6.1.1 Communication between the Physical Page and the Digital Page 

31 The user can interact with a coded substrate page by positioning the pointer on the page 

32 and clicking a button on the pointer. The pointer recovers its position relative to the page frame. 

33 The coded substrate communication infrastructure, described elsewhere, delivers the 

34 position and the clicking events from the pointer to the digital page, delivers output from the digital 

35 page to a peripheral whose address has been specified, and delivers mode-switching commands 

36 from the digital page to the pointer. Communication between the physical page and digital page is 

37 limited to these kinds of information. 

38 A digital page could do arbitrary computation with arbitrary data. Although a digital page's 

39 capabilities are not restricted, there are certain capabilities that make sense for a digital page, that 
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1 will be useful across a wide range of coded substrate documents, and for which support can be 

2 provided in a generic manner; several of these are described here. 

3 To reiterate, a digital page receives positions (x, y coordinates) and produces output (which 

4 may take the form of instructions to run a program on another machine, and input data for that 

5 program). The coordinates could be delivered to the digital page in any appropriate way, and the 

6 output could similarly be delivered to the machine on which it is to be displayed or executed in any 

7 appropriate way, as described elsewhere. 

8 A basic coded substrate capability is to initiate an action by clicking in an active region on 

9 the page. Each active region is typically graphically marked, as with icons, labelled boxes, or 

10 highlighted words. Active regions have two elements: regions and actions. 

11 6.1.2 Regions 

12 A digital page receives the position of the pointer, and must determine if that position is 

13 within an active region. To this end, it can have a description of the active regions on the page. 

14 Rectangular regions have a particularly simple description. Given two diagonally opposing 

15 corners of a rectangle, a digital page can determine whether a given point lies within it or not, and 

16 the function used to tell is also particularly simple. 

17 Arbitrarily complex regions can be described in such a manner that a digital page can 

18 determine whether a given point lies within it. The function needed to decide may be complex, and 

19 different types of descriptions require different functions. We will call a pair consisting of a 

20 description of a region and a function to decide whether a point lies in it a "region". 

21 6.1.3 Actions 

22 Once the digital page has determined that a position is within an active region, it can cause 

23 an action (or a number of actions) to be executed. For this purpose, it can have a description of the 

24 action(s) for that region. Possible ways to describe actions include (but aren't limited to) programs 

25 (or functions), with or without input, pointers to programs (internal or external to the digital page), 

26 with or without input, and (type, data) pairs, where the type element identifies a conventional action 

27 to take, and the data element is input (e.g., URL, http://www.xerox.com could mean "open the web 

28 page http://www.xerox.com in a browser"). 

29 6.1 .4 Action Tables 

30 An action table is a table of (region, action) pairs, where the action element of the pair may 

31 describe a sequence of actions. A digital page can have at least one action table, describing active 

32 regions on the page. At any time, one action table can be active, which is to say that positions 

33 delivered to the digital page will be looked up in that table. 

34 6.1 .5 The Output Player 

35 A coded substrate-based action (one arising through some user interaction with a coded 

36 substrate document using a pointer) often results in output on a machine near the user, while the 

37 digital page resides at some arbitrary location on the network. This is why the output of a digital 

38 page may be input to a program to be run on another machine. It happens that, in the above 

39 examples, Ariane's local peripheral is her TV screen. In this case, output takes the form of a video 
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1 signal sent to her TV. But if her peripheral were a computer, the output would need to be of a 

2 different form, and in fact, of a different form for each different computer platform. The digital page 

3 should not be encumbered with the details of the form of the output. Each peripheral can be 

4 equipped with an output v player', so called because it knows how to play various types of data; 

5 sound data is played on the audio system, video is played with a video player, text is displayed in a 

6 conventional way on the screen. Current browsers provide such capabilities in a platform- 

7 independent way for computers. The peripheral can run an output player directly or the peripheral 

8 can be controlled by a computer that runs an output player that sends output of the right form to the 

9 peripheral. 

10 An output player receives typed data. Video, audio, and text are types that have already 

11 been mentioned; others could include control information (volume, colour, and so forth), 

12 spreadsheet position and value, etc. 

13 In addition to giving the digital page platform independence, the output player solves the 

14 problem of how to push information and actions to another machine. A program on one machine 

15 does not routinely have permission to display data or execute a program on another machine. In 

16 the case of coded substrates, the peripheral near the user can be registered with the 

17 communication infrastructure. Either as part of the registration process or explicitly by the user at 

18 the point of interacting with a coded substrate document, an output player is launched on the 

19 peripheral. The digital page can send data to the output player, whose address is known to it by 

20 virtue of the registration; it need not launch a program on the peripheral. The output player, in 

21 contrast, can launch programs on the peripheral, subject to local permissions. An interesting kind of 

22 audio output would be to a mobile telephone to which the pointer is connected. For instance a 

23 tourist guide printed on a coded substrate could be used to send audio output to the user on his 

24 mobile phone giving him information about theater programs, etc. 

25 6.1 .6 Communication with Applications 

26 In the course of executing actions, a digital page may launch other applications. When a 

27 digital page launches an application with which it may have continuing interaction, it maintains a 

28 connection with the application (such as a socket). The digital page may launch an application (for 

29 example, a browser) on the user's peripheral via the output player. The output player returns to the 

30 digital page the information it needs (such as a socket name) to establish a direct connection with 

31 the application. 

32 A digital page may also provide a mechanism to allow the user to establish a connection 

33 between itself and an already-running application. In other words, by interacting with the coded 

34 substrate, the user can give inputs to or receive outputs from a running application that is external to 

35 the digital page and not produced in conjunction with a printed document. 

36 Also, results of one action, after being locally stored, can be input to another. 

37 6.2 Output Actions 

38 Here we describe one of a variety of available ways to implement the output actions 

39 attributed to a user such as Ariane. 
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1 6.2.1 Action 1 : Text Output 

2 Ariane is in doubt about the meaning of a word. She clicks on it; an explanatory note for the 

3 word is displayed on her TV screen. 

4 The digital page is responsible for producing the explanatory note; how the note gets to 

5 Ariane's TV screen is the responsibility of the communication infrastructure. 

6 The active action table contains a pair (W, E), where W is the region containing the word, 

7 and E is, for example (Output, Exp), where Output is a type meaning "display the data element of 

8 the action pair to the user" and Exp is the text of the explanation. 

9 The digital page receives the position of Ariane's pointer. It looks up the position in the 

10 active action table and determines that it is in region W. it therefore executes action E, that is, 

1 1 outputs Exp, which is sent to the output player of Ariane's TV. 

12 Note that, for this mechanism, the explanatory note can be provided by the publisher of the 

13 document a priori, and the word can be graphically identified as one that has an explanation. The 

14 ability to look up a word in an on-line dictionary would provide a definition for any word, without 

15 putting a burden on the publisher. Of course, the dictionary definition may not be identical to the 

16 hand-crafted explanatory note, which could take advantage of context. But in a great many cases, a 

17 dictionary definition would be valuable. In order to provide this, a general text-selection mechanism 

18 as described below in Section 6.5 can be used. 

19 6.2.2 Action 2: Audio Output; Action Table Switching 

20 Ariane clicks on the Recite box, then on the beginning of a line; the line is recited by 

21 Laurence Olivier over her TV speakers. The active action table contains an entry for the Recite box. 

22 Its action is of a different type; it affects the state of the digital page process. It makes the active 

23 action table inactive, and makes some other action table active— one that contains an entry for each 

24 line, as follows. The region element of an entry describes the region containing the line. The action 

25 entry has two parts. The first part says to output audio data (stored as part of the action), which is 

26 the recitation of the line by a famous actor. The second part restores the previous action table as 

27 the active action table. 

28 Note that this mechanism of action tables, and changing active action tables, is only one 

29 implementation of many. Another general mechanism would be to associate operators and 

30 operands with regions, distinguishing between the types of objects. Action 1 , Text Output, would 

31 bundle the necessary operand with the operator, so only one click is necessary. For Action 2, one 

32 click would designate the reciting operation and a second click would designate the line as its 

33 operand. 

34 This also could be done via text selection as opposed to a priori definition of the regions 

35 and provision of the recitations. It would require a reading of the play aligned to the text and 

36 segmented into lines; then selection of arbitrary text could trigger location of the point in the reading, 

37 the audio data from there to the end of the segment could be output. 
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6.2.3 Action 3: Video Output 

Ariane clicks on the Play Scene box, then on the beginning of a scene, and a video of 
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Orson Welles 1 interpretation of the scene is played on her TV. 

This is just like the previous case, but the action table for the Play Scene box has entries 
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appropriate to playing videos of scenes. 

6.3 Annotating; Adding Actions 



7 Various techniques can be used to enable a user to add links and actions to an already 

8 published page. 

9 jhe example is the following: Ariane has previously connected to the Web pages provided 

10 with the coded substrate document that she's reading (Othello), by clicking on the Main Web Page 

11 box. As a result, she has a browser on her screen, open to one of these Web pages. She uses a 

12 normal pencil to make a note to herself on the page. She clicks on the Link To Web Page box. 

13 She clicks on the note she just wrote. Henceforth, clicking on her note causes the Web page to pop 

14 up on the screen. 

15 a similar but slightly more general example is easier to illustrate, after which we'll return to 

16 the above. Consider: Ariane has a browser on her screen, open to an arbitrary Web page. She 

17 uses a normal pencil to make a note to herself on the page. She clicks on the Link To Web Page 

18 box. She clicks on the note she just wrote. A dialog box pops up on the screen. She copies the 

19 URL of the Web page from the browser to the dialog box and clicks "Apply" in the dialog box. 

20 Henceforth, clicking on her note causes the Web page to pop up on the screen. 

21 The active action table contains an entry for the Link To Web Page box. Its first action 

22 changes the mode of the digital page. Rather than continuing in its normal event-driven mode, the 

23 digital page waits for the next click from the user. When it's received, the position P of the pointer is 

24 recorded, rather than being interpreted as a standard interaction (it is not looked up in the active 

25 action table, as would normally be the case). A region R of some conventional size with P at its 

26 centre is created. The above-described dialog box interaction is initiated, and the URL U given by 

27 the user is used as the data element of an action URL, U. The entry R, URL, U is added to the 

28 active action table. When the user subsequently clicks in the defined region, the standard action- 

29 table mechanisms will pop the Web page up on the screen. 

30 An Add Action box would function in a very similar manner, the difference being that the 

31 digital page would not automatically interpret the data it gets from the dialog box as a URL. Instead, 

32 the user must provide a complete description of the action. For example, assuming actions are 

33 described as type, data pairs, the user could select the type from a menu and enter the data. A type 

34 "Program" that launches a program would provide a catch-all to allow virtually any type of action to 

35 be added. 

36 In a further example, Ariane doesn't have to type in the URL because the digital page 

37 launched her browser, and thus can communicate directly with it. The browser provides its current 

38 URL when queried. Browsers that don't already have this functionality can be extended to provide 
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1 it. Thus the dialog box interaction with the user can be replaced with an interaction between the 

2 digital page and the browser, while the rest of the scenario remains the same. 

3 6.4 Using the Pointer as a Mouse 

4 The pointer can be used as a mouse, and, more specifically, positional information at 

5 regular intervals can be treated as cursor ositions. 

6 The example is: Ariane clicks on the Mouse Mode box and subsequently uses the pointer 

7 as a mouse, with the page as the mouse-pad. 

8 The example can be modified so that the Mouse Mode box acts as a toggle; if she clicks on 

9 Mouse Mode when in Mouse Mode, the pointer returns to its non-Mouse-Mode mode of operation. 

10 Meanwhile, an application is running, displayed on Ariane's TV (more generally, on the user's local 

11 peripheral). The digital page has a connection to this application, either explicitly established by the 

12 user or because the application was launched by the digital page. 

13 The standard mode of the pointer is to initiate communication with the digital page each 

14 time the user clicks, sending its current position and the click event. Thus in non-Mouse-Mode 

15 operation, the digital page gets the position of the pointer whenever the user clicks. 

16 In Mouse Mode, the digital page gets the position of the pointer at regular intervals, without 

17 the user clicking (and with no click event transmitted). When the user does click, the click event 

18 and position are sent as usual. 

19 How It's Done. The active action table contains an entry for the Mouse Mode box. Its 

20 actions follow: A command is sent to the pointer, to change its mode of operation. In the new 

21 mode, the pointer initiates communication every n timer ticks, sending the digital page its position 

22 each time, in addition to sending its position with a click event when the user clicks. This change of 

23 mode, and thus the need for the pointer to receive control information from the digital page, can be 

24 avoided at the cost of more communication traffic, by having the pointer always send its position 

25 every n ticks. In non-Mouse-Mode, the digital page would ignore positional information that is not 

26 accompanied by a click event. Alternatively, the pointer could distinguish itself whether it's in 

27 mouse-mode or non-mouse mode (e.g., the button is held down during mouse mode, or 

28 alternatively, positions are sent every n timer ticks when the pointer is moved in contact with the 

29 page). 

30 The mode of the digital page process is changed, too. The first position RP that the digital 

31 page receives with no accompanying mouse click is recorded. A cursor is displayed on the output 

32 screen, because cursor display is one of the capabilities of the output player, and its position CP on 

33 the screen is recorded. Each time a position P is received, the difference between P and RP is 

34 computed. That difference is added to CP and the cursor is displayed at the new value of CP. RP 

35 is replaced by P, and the process iterates. 

36 When the digital page receives a position P with a mouse click event, it updates CP and RP 

37 as usual, and in addition it forwards the mouse click event, with CP substituted for P, to the 

38 application. By "forwards the mouse click event" we mean that the digital page puts the same type 

39 of mouse click event (changing the position) in the application's event stream. 
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1 Of course, if the application is the digital page itself, these issues are absent. 

2 Absolute Positioning. What has just been described is how to use positional information 

3 relative to the printed page frame to simulate a standard mouse. As a positioning interface, this has 

4 the advantage of being familiar to mouse users. 

5 This is relative positioning: the user moves the pointer relative to the current position of the 

6 cursor, and the absolute position of the pointer on the page is irrelevant. 

7 An alternative is absolute positioning. Here the user moves the pointer relative to the page 

8 frame; the absolute position of the pointer is crucial. If the user wants the cursor up near the top 

9 right-hand corner of the screen, the user can put the pointer near the top right-hand corner of the 
10 page. 

n |f the page is the same size and shape as the screen, then the absolute position of the 

12 pointer relative to the page frame could be directly translated to the screen; this combines elements 

13 of using a mouse and pointing at a touch-screen. 

14 The printed page is rarely the same size and shape as the screen. However, the page 

15 dimensions can be scaled to match the screen (or to match some window on the screen), and the 

16 position on the page can be interpreted according to the scaled dimensions as a position on the 

17 screen (or in the window). 

18 6.5 Selecting Content 

19 The user can use the pointer to select content from a printed page and make the content 

20 available as input to an action or application. 

21 Section 6.2.1 explained that, in order to provide an explanatory note about a word, the note 

22 can be stored in the action table entry for the region around the word. The author can decide to 

23 provide a note for that word as part of the publication process. It is problematic to support cut and 

24 paste from a coded substrate page into another document, because one can't know ahead of time 

25 what chunks of text the user might want to cut. To help a foreign reader understand the document, 

26 for example, by allowing her to display a translation of some part of the text, it would not be feasible 

27 to translate the document into all languages. Machine translation could be offered as one option, 

28 but it cannot be known ahead of time what part of the text she'll need help with. 

29 For a complete solution, isomorphism could be maintained between the physical page and 

30 the digital page. But simple text selection can be done using action tables. 

31 a user may, for example, select a word or an object by double clicking on it. 

32 This can be supported with an action table with an entry for each word and each selectable 

33 object. The region element of the entry describes the region bounding the word or object, and the 

34 action element is (Select, W), where W is the word or object, and the conventional selection action 

35 is to copy W to a clipboard. 

36 The digital page uses this action table whenever it receives double clicks. The digital page 

37 can distinguish double clicks as such if a time-stamp is sent with each click event, as transmission 

38 limes from the pointer to the digital page may not be uniform. Or the pointer could recognise the 

39 click type. 
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1 Another selection method, such as triple clicking to select a line, could be added by adding 

2 an action table with an entry for each line. 

3 Finally, another common text selection method is to push the button down, drag the pointer, 

4 and release the button elsewhere. The digital page receives the position of the pointer at the two 

5 locations. 

6 This selection method is not efficiently amenable to an action-table implementation (without 

7 a separate representation of content and a mapping from positions into that representation). An 

8 implementation analogous to the ones just described would require entries for all pairs of 

9 characters. 

10 An example illustrates the concept of selecting content. The example, in which Ariane 

11 wants to have the word ^exsufflicate" translated into French, is as follows: 

12 She clicks on the box Translations on the top of the physical page. A special menu appears 

13 on the screen, giving a choice of several languages. She selects French. She now has a choice 

14 between either a word-in-context automatic translation or a number of standard literary translations. 

15 She selects Gide's translation, then directs the pointer to the difficult word. Gide's translation of the 

16 line appears on the screen, with the word(s) corresponding to ^exsufflicate" singled out in boldface. 

17 Note that the entire line is translated, though the user pointed to a single word. 

18 This can be done with action tables as follows: The Translations box has an entry in the 

19 active action table, or its region is type Action; either way the action puts up the menus and collects 

20 the user's decisions. The digital page has access to Gide's translation, which is aligned sentence- 

21 for-sentence, and within each sentence, word for word, with Ariane's copy of Othello. Given a 

22 sentence number and the number of a word within the sentence, the translation function finds the 

23 corresponding sentence and word (or words) in the translated text. It boldfaces the word and 

24 returns the translation. 

25 The sentence and word numbers cam be [rpvoded. for text identification, by an action table 

26 built specifically to support this function. It has an entry per word, containing the sentence number 

27 and word number. 

28 6.6 Isomorphism 

29 Coded substrate technology as described herein allows a person to interact with paper in 

30 much the same way as one could interact with a computer screen. A digital representation of an 

31 image that is isomorphic to the image printed on a page can enable a position on a printed page to 

32 be used as input to any computer application that can accept input in the form of position in a digital 

33 representation of an image. Examples include browsers, word processor, and substantially any 

34 application with a graphical user interface or for which a graphical user interface could be built. 

35 Related techniques are described in copending, coassigned U.S. Patent Application No. 

36 09/233,882, entitled "Multi-level Selection Methods and Apparatus Using Context Identification for 

37 Embedded Data Graphical User Interfaces", incorporated herein by reference. 
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1 As described above, the position of the pointer on the page can be delivered to the digital 

2 page. This positional data is analogous to the position of the mouse on the screen, which is the 

3 information that a standard computer application receives. 

4 The mechanisms for implementing coded substrate technology actions differ from the 

5 mechanisms one would use for standard computer applications, notably due to the fact that the 

6 paper does not change as a screen does. 

7 Two kinds of mechanisms could be used to implement actions such as Ariane's as 

8 described herein. In one, the complete image and content of the printed page is represented, as 

9 such, in the digital page, while in the other, only the actions are represented in the digital page. The 

10 latter is functionally weaker than the former, because its actions can only use content that has been 

11 associated with the action in advance. Arbitrary text selection is a case in point; it requires a 

12 representation of the content in the digital page. 

13 The former is referred to herein as isomorphism between the physical and digital page. 

14 Isomorphism is illustrated in Figs. 14 and 15. Fig. 14 shows isomorphism in which selection of an 

15 entity on the physical page by positioning a pointer is coupled with selection of the counterpart entity 

16 in the digital page. Fig. 15 shows isomorphism in which using a pointer to indicate a destination 

17 position to which a selected entity is moved or copied is coupled with an indicating in the digital 

18 page of the counterpart position as a destination. 

19 it is common practice to map from a position on the screen to a position in a digital 

20 document that has been rendered on the screen; this is what is done by browsers and text editors, 

21 for example. 

22 To make use of a representation of the content of a printed page, it must be possible to 

23 map from a position on the page to a position in the digital representation. 

24 This can be done by using a digital representation that is isomorphic to the printed page. In 

25 this way, positions on the page match one-for-one positions on the screen. 

26 One can imagine the printed page reproduced on a screen. As the pointer is moved on the 

27 physical page, the cursor moves, simultaneously (modulo network delays) and in parallel. Button 

28 clicks on the pointer are treated as clicks on the mouse, as suggested in Fig. 15. The digital page 

29 process acts just as it would if the user were moving the cursor, and clicking, with the mouse. 

30 Given this kind of parallelism, standard computer application techniques (such as those 

31 used in editors and browsers) may be used to implement Ariane's actions. Provision of support for 

32 those actions can be made using techniques known in the art. 

33 This parallelism can be achieved by associating the digital page process with a special 

34 purpose window manager. This window manager manages the window displaying the page. The 

35 window size is identical to the page size. The window manager takes positional information from 

36 the pointer rather than the mouse— but this is transparent to the digital page process. 

37 There is no need for this window to be rendered to a screen— it can be entirely virtual. 

38 The isomorphism is between the image on the physical page and one aspect of the digital 

39 page: a digital representation of that image (or information from which such a representation can be 
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1 constructed). The digital page contains more information, including the mapping from the image to 

2 the content and the actions to be performed. 

3 Constructing a digital representation that looks like what you get when you print to paper 

4 was a difficult problem — most text editors and browsers display a document that differs on-screen to 

5 one degree or another from the printed version. Adobe PDF format provides page-description 

6 functionalities that are strong enough that the printed and digital representations truly look alike. 

7 Other page description formats may provide similarly. 

8 A special-purpose window manager is just one way among many to take advantage of such 

9 an isomorphism. Another way would be to use a standard window manager, but to modify the 

10 positional and event information as it comes from the pointer so that it appears to the window 

11 manager to be a mouse event. This moves some of the special-purpose processing, and some 

12 elements of window-management, into the communication infrastructure, which would communicate 

13 with the window manager. Alternatively, positions can be interpreted directly by the digital page 

14 process. Window management per se (which window is on top, which application handles an 

15 event) isn't relevant. 

16 But an event handler, which is typically provided by the window manager, would have to be 

17 incorporated into the digital page. 

18 7. Special Applications 

19 7.1 Writer-pointer. 

20 Here, using the pointer disclosed in Fig. 1 1 which writes on coded substrates, the position 

21 of the pointer is monitored during hand-writing through the usual localisation device. This permits 

22 the marks to be registered in digital form simultaneously to their physical production by the writer, in 

23 a transparent way. Strokes made with the pointer can therefore be recognized and recorded for 

24 further use. This provides the following applications: 

25 - drawing on a coded substrate and getting a digital copy of the drawing, on which 

26 various operations can be later performed (smoothing, shape-recognition, etc.) As the pen 

27 movements are recorded in time, such as by associating timestamps with movements, the digital 

28 drawing is dynamic, as opposed to the case of scanning the final state of a drawing. This dynamicity 

29 might help a shape-recognition program identify a circle, a square, etc. There are also applications 

30 in signature authentification, by making the dynamics of the strokes a part of the signature. 

31 - taking hand-written notes during a meeting, which are later accessible in digital 

32 form; dynamicity may help perform hand-writing recognition, if an ascii representation of the text is 

33 desired. 

34 - writing (or drawing) on a flip-chart, and keeping a digital copy; or writing with a 

35 marker on a white-board having a coded substrate, and keeping a digital copy. 

36 - filling multi-choice paper forms. 

37 - putting by hand editor's marks on a draft typescript and having these marks 

38 transmitted and interpreted on the digital side in order to produce a corrected version. 
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1 The writer-pointer could be implemented with a writing mode, in which the image containing 

2 positional information is transmitted at frequent regular intervals, and can be used to determine the 

3 position of the point. Interpolation may be used to derive a continuous or approximately continuous 

4 line in the coordinate space of the page, which may be recorded as a stroke on the digital page, as 

5 a series of positions or as a function derived from them using conventional drawing program 

6 techniques; alternatively, the digital representation could be modified or annotated. 

7 7.2 Controlling access rights I 

8 As has been discussed, coded substrates have a number of applications relating to 

9 interacting with services. As will be seen, they also have significant potential for control of access 

10 rights to data as well as to digital commerce. 

11 Consider the representative case of a classical recording publishing company, Digital 

12 Grammophon, or DG; this company distributes musical recordings over the net, and it must find 

13 ways to have listeners pay for them. Coded substrate can help to solve this problem. 

14 DG sells, through record or book shops, listening cards printed on coded substrate. One 

15 such card, for instance, lists the last six Mozart symphonies. Each movement is listed as a 

16 subheading under the corresponding symphony heading. The customer buys the card, goes back 

17 home, clicks on the heading for the 40th, and the music is played for him on his hi-fi peripheral 

18 device. 

19 This scheme is not commercially secure for the DG company with respect to an 

20 unscrupulous user intent on defeating copyrights: 

21 • Such a user might legally buy a listening card for the symphonies and reproduce, 

22 through standard photocopying, its printed content on thousands of cards, and distribute these 

23 through parallel channels. Of course, this scheme will not work. A buyer of such a card will not be 

24 able to use it in any way. It is instructive to compare this situation with the case where the cards 

25 produced by DG would be printed on conventional paper, and where marks printed on these cards, 

26 such as barcodes, would encode the addresses of the music tracks. In this case, a standard 

27 photocopy of the original card would be functionally indistinguishable from this original card. 

28 • Such a user might become more sophisticated and produce perfect copies of the 

29 original card, including the coded substrate (an operation, which, by the way, may be illegal if coded 

30 substrates were a monopoly of a coded substrate supplier). 

31 Although such an operation would be technically difficult and costly, it might not be 

32 impossible. However, DG can easily prevent such a problem. Each piece of coded substrate has a 

33 unique identification number page-id, which is associated at the DG site with the digital 

34 representation of the page (this is of course a many-to-one mapping). Because of this unique 

35 identification for each physical card, DG can detect two simultaneous access attempts through the 

36 same page-id, and knows that this event corresponds to an illegal situation. This is an example of a 

37 "valid request criterion", meaning a criterion that can be applied to distinguish valid requests from 

38 invalid requests. 
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1 Upon detecting an invalid request, DG can then take measures of different degrees of 

2 severity. A drastic measure would be to void the guilty page-id of any future access rights. Under 

3 such a scheme, people buying the illegal copies would be quickly discouraged. 

4 Other less severe schemes could be implemented, such as putting limits on how many 

5 cumulative hours a given page-id could be used each day, or on the total number of times it could 

6 be used in its lifetime. These schemes would obviously have broader commercial implications than 

7 just preventing piracy: for example, one could buy the right to listen 10 times to the "Jupiter" 

8 symphony. 

9 7.3 Controlling access rights II 

10 But where the user is not to be discouraged by the measures in section 7.2 above, he may 

1 1 make a further attempt to cheat the DG company. 

12 Suppose the user has noticed that all the DG cards for the Mozart symphonies bear 

13 consecutive page-id numbers. From buying one such card, she can therefore predict what the 

14 thousands of DG cards for these recordings look like. She then goes on and produces cards that 

15 have the right page-id numbers on them. 

16 Now this move is potentially dangerous for DG. There is no obvious valid request criterion 

17 that it can apply to distinguish a card bought legally from a card bought from the unscrupulous user. 

18 But there is a radical counter-measure that DG, with support from the coded substrate 

19 supplier, can take. The coded substrate supplier, rather than simply encoding a page-id number in 

20 the white coded substrate sheets it sells to DG, now pairs the page-id with a hash-code page-id- 

21 code. The function mapping a page-id into the page-id-code is a secret property of the coded 

22 substrate supplier; it may be implemented in a variety of ways: through a random number generator, 

23 a secret hash-coding algorithm, or a public-key-cryptography scheme. Although there are some 

24 differences between these various approaches, what is crucial for the current problem is the 

25 following thing: the page-id-code cannot be predicted from the page-id by a person outside of the 

26 coded substrate supplier. On the other hand, the coded substrate supplier, when selling coded 

27 substrates sheets to DG, provides DG with a list of <page-id,page-id-code> pairs, so that DG knows 

28 the code associated with any sheet of coded substrates it owns and can apply a corresponding valid 

29 request criterion. 

30 Now the unscrupulous user, even if he is able to predict which page-id numbers will be used 

31 for the Mozart symphonies, has no way to know the page-id-code numbers associated with these 

32 pages. The only way she could know these numbers would be by buying all the DG cards 

33 corresponding to the symphonies, a somewhat uncongenial prospect. 

34 We thus see that, while coded substrates may still be copiable, given the aforementioned 

35 additional technology, with a page-identifier coding scheme, it becomes unforgeable: in order to 

36 produce a usable card, the pirate needs to own an original for that card. 

37 It should be noted here that the idea of privately encoding the page identifiers does not 

38 strictly require the coded substrate medium. It could also be implemented via conventional printed 

39 marks on the card. Although this might already be an interesting possibility En itself, the coded 
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1 substrate medium fits the bill very well here, because one generic architecture (paper, pointer, net 

2 infrastructure), already employed in several other functions, can be readily adapted to the case at 

3 hand. The only modification of the original scheme consists in adding a page-id-code to the page-id 

4 during the production phase at the coded substrate supplier, an operation with minimal 

5 supplementary cost. 

6 Another access rights example is as follows. The user buys stamp at local news-stand, 

7 transfers its code to mail-order house, mail-order house transfers code to the bank. 

8 Suppose the coded substrate supplier, under the supervision of a commercial bank, 

9 produces small pieces of adhesive coded substrates, shaped like post stamps, and printed in 

10 various denominations (10cts,1$,10$, etc.). These stamps are sold, under a sealed wrapper, at the 

11 bank offices, through local newsstands, etc. Using coded substrate technology, the stamps are 

12 provided with a unique page-id and with a corresponding page-id-code as described above. The 

13 pair <page-id, page-id-code > is known only to the bank. 

14 A customer wishes to purchase some catalogue item through a mail-order company. She 

15 buys enough stamps to cover her order, sticks them on an empty space provided next to the item in 

16 the catalogue, and then, using the pointer, clicks first on the item, then on each of the stamps. The 

17 action of clicking on a stamp has the effect of transferring the <page-id , page-id-code > pair to the 

18 mail-order company. The mail-order company can then transfer these codes to the emitting bank, 

19 which credits the mail-order company's account for an amount equal to the stamp denomination. 

20 The bank will accept an operation with a given <page-id , page-id-code > pair only once: Any 

21 attempt to re-use this pair for a later operation will be rejected as failing to meet the applicable valid 

22 request criterion, so that each stamp can effectively be used for payment only once in its lifetime. 

23 After valid payment has been confirmed by the bank to the mail-order company, the item is sent to 

24 the customer. 

25 The procedure just sketched uses simple coded substrate technology, a form of 

26 commercial transaction - which preserves the anonymity of the buyer of stamps - can be provided 

27 relatively easily. 

28 Variations 

29 The implementations described above could be varied in many ways within the scope of the 

30 invention. 

31 Implementations described above use DataGlyphs to encode action/medium identifiers, but 

32 various other machine-readable markings could be used. For visually nonobstructive markings, it is 

33 desirable to have markings for which the shape of a pattern or collection of markings appears 

34 uniform to the eye under normal viewing conditions even if one of the markings changes value. 

35 Implementations described above use pointers that include cameras or handheld scanners 

36 to obtain input signals that include information about machine-readable markings, but any other 

37 appropriate detection devices and circuitry could be used, including detection devices that are not 

38 handheld. 
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1 Implementations described above provide input signals to processing circuitry when they are 

2 received and actions are produced immediately, but the invention could be implemented in a way 

3 that stores input signals from one or more scans in the pointer temporarily, possibly in a 

4 compressed form. Later, the pointer is docked to a host, which could be a computer, a phone, a fax 

5 machine, or the like, and the stored input signals are provided for decoding by the host's processor 

6 and production of actions without the need of communicating the stored signals immediately to a 

7 network, and therefore permitting simpler or cheaper input devices. 

8 Implementations described above produce actions through the Internet, but the invention 

9 could be implemented to produce actions through any suitable network. 

10 The implementations described above use marking media that are sheets of paper, but any 

11 other appropriate marking media could be used. 

12 Implementations described above use action/medium identifiers of several different types, 

13 including globally unique page identifiers and sticker identifiers, page identifiers paired with location 

14 identifiers, document identifiers paired with action identifiers, and so forth. Various other 

15 action/medium identifiers could be used within the scope of the invention, including, for example, 

16 network-wide page identifiers. 

17 Implementations described above use obtain action identifiers of several types from 

18 action/medium identifiers, including taking part or all of the action/medium identifier as the action 

19 identifier, but action identifiers could be obtained in various other ways, including decoding. 

20 Implementations described above obtain network addresses for peripherals using sticker 

21 identifiers provided to a router, but network addresses for peripherals could be obtained in various 

22 other ways. Similarly, network addresses for an action device such as a server could be obtained in 

23 various ways other than those described above. 

24 Implementations described above involve printed documents on coded substrates, but the 

25 invention could be implemented with marking media that are not coded and with documents that are 

26 handwritten or produced in some way other than printing. 

27 Implementations described above attempt to maintain isomorphism between a physical 

28 page and a digital page, but the invention can be implemented without isomorphism, and could be 

29 used to obtain any action that can be produced through a network. 

30 It will be appreciated that the techniques described herein may also be used in conjunction 

31 with the related techniques described in copending, coassigned U.S. Patent Application Nos. 

32 09/BBB,BBB (Atty. Docket No. R/98004), entitled "Obtaining and Using Data Associating Annotating 

33 Activities With Portions of Recordings", and 09/CCC,CCC (Atty. Docket No. R/98005) entitled 

34 "Obtaining Network Addresses from Identifiers", both incorporated herein by reference. 

35 In implementations described above, specific acts are performed that could be omitted or 

36 performed differently. 

37 In implementations described above, acts are performed in an order that could be 

38 modified in some cases. 
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1 Implementations described above use currently available machine-readable marking, 

2 detection, computing, and networking techniques, but could readily be modified to use newly 

3 discovered techniques as they become available. 

4 Applications 

5 The invention could be applied in various ways. 

6 The invention could be applied in a synchronous mode to provide interactive books, 

7 magazines, maps, pocket encyclopedias, product catalogues, examination forms, paper address 

8 books, and so forth. 

9 The invention could be applied in an asynchronous mode to allow collection of bookmarks 

10 while reading a document such as a newspaper or magazine, after which the bookmarks could be 

11 used in a batch to retrieve email clippings or print additional information. 

12 Because the pointer behaves like a paper mouse, it can be used to record manual 

13 movements in real time, such as drawing or writing motions. Handwritten notes taken during a 

14 meeting or during making or playing of a recording can be captured and processed, handwritten 

15 faxes can be sent without using a computer, and free-form information requests can be written in an 

16 input rectangle inside an advertisement and transmitted to the sponsor for feedback. 

17 Miscellaneous 

18 The further description of features of the invention set forth in Dymetman, M., and 

19 Copperman, M., "Intelligent Paper", in Proceedings of Electronic Publishing '98, April 1998, Saint 

20 Malo, France, Springer- Verlag, 1 998, pp. 392-406, is incorporated herein by reference. 

21 Although the invention has been described in relation to various implementations, together 

22 with modifications, variations, and extensions thereof, other implementations, modifications, 

23 variations, and extensions are within the scope of the invention. The invention is therefore not 

24 limited by the description contained herein or by the drawings, but only by the claims. 
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